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REMARKS 

2 Claims 43-44 were previously withdrawn in response to an earlier 

3 restriction requirement 

4 Claims 1-42 remain in this application, 

5 Applicants* previous arguments were persuasive over the Office's previous 

6 cited art, and the Office has raised new grounds of objection based on new art that 

7 is considered for the first time in this response, namely U.S» Patent 6,557,042 to 
He et al. 

9 

10 35 U,S.C. 8103 

n Claims 1-^9 

12 Claims 1-9 are rejected under 35 U.S.C. §103(a) as being unpatentable over 

13 Atsushi Ono et al, "A Flexible Content-Based Image Retrieval System with 

14 Combined Scene Description Keyword", page 201-208, June 17-23, 1996, 

15 Multimedia Computing and System (hereinafter "Ono) in view of U.S. Patent 

16 6,557,042 to He et al (hereinafter ^He")* Applicants respectfully traverse the 

17 rejection. 

18 Independent claim 1 recites; 



19 
20 



22 



23 



24 



A method comprising: 



identifying, in response to a search query, first multimedia objects 
21 having an associated keyword that matches a keyword in the search query 

and second multimedia objects that have content features similar to those of 
the first multimedia objects; 



presenting the first and second multimedia objects to a user; 



monitoring feedback from the user as to which of the first and 
second multimedia objects are relevant to the search query; and 
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^ annotating one or more of the multimedia objects, which are deemed 

2 relevant by the user, with the keyword. 

The combination of Ono and He fails to teach or suggest the method of 
claim 1. Ono describes searching for images based on keywords. The image 
search is based exclusively on keywords associated with the images. Ono relies on 
automated keyword extraction based on scene or image recognition. In other 
words, based on a scene or image recognition, a keyword associated with the 
image* The extracted keywords are then used to retrieve images. All multimedia 
objects (images) in Ono are identified by particular keywords associated with the 
multimedia objects (images). In Figs. 4 and 5 of Ono, for example, the images are 
retrieved using two unique keywords associated with each image. "The keyword 
enables us to retrieve images by not only referring to the meaning of the image but 
also using attributes such as location, color, shape of segments, and relation 
between segments or objects". Ono at page 202, left col., Iinesl2-15* 

Because Ono exclusively relies on keywords, Ono does not teach or suggest 
"identifying, in response to a search query, first multimedia objects having an 
associated keyword that matches a keyword in the search query and second 
multimedia objects that have content features similar to those of the first 
multimedia objects ". The Examiner cites Ono at page 202, left col., lines 15-22 as 
teaching this element; however, this section of Ono recites "[a]nd when retrieving, 
the use of a recognition model which is used to give the kejword can offset the 
incompleteness of the present image recognition technique and improve retrieval". 
Ono shows that retrieving (see section 3 of Ono) and the recognition model (see 
section 4 of Ono) are based on keywords. Ono does not teach or suggest 
'^identifying, in response to a search query ... second multimedia objects that have 
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content features similar to those of the first multimedia objects" as recited in the 
claim, since the identifying or retrieving as disclosed in Ono is based exclusively 
on keywords. 

The Examiner also cites Ono page 202, left coL, Imes 15-22 as teaching the 
element of **presenting the first and second mxiltimedia objects to a user." As 
discussed the second images (multimedia objects) are retrieved (i.e., identified) 
based on similar content features, and Ono relies on retrieving based on keywords. 
Therefore such second multimedia objects are not presented to the user. 

The Examiner relies on He as teaching the elements of '^monitoring 
feedback from the user as to which of the first and second multimedia objects are 
relevant to the search query" and "annotating one or more of the multimedia 
objects, which are deemed relevant by the user, with the keyword". The Examiner 
particularly argues that: 



He teaches the annotating process using button 332 of Fig 5 
excepting (sic, except) for annotating the multimedia objects based 
on eh the feedback. He teaches "there are may be a relationship 
defined between a marketing group and an engineering group, such 
that user feedback from a user in the marketing group also modifies 
the corresponding segment score in the engineering group • This 
modification may the same (e.g. increase both scores by two for 
positive feedback), or different (e-g., for positive feedback increase 
the segment score in the user*s group by two and increase the 
segment score in the related group by one" (col. 9, lines 56-64). This 
teaches the increasing the score by two when there are positive 
feedbacks from the user means annotating the segment by two of the 
marketing and engineering group. Therefore, it would have been 
obvious to one ordinary skill in the art at the time of the invention 
was made to include the user's feedback to modify the scores to 
annotate the marketing and engineering group to the multimedia 
object segment to allow the retrieval of the same segment that 
relevant to these marketing and engineering. 
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The Examiner does not discuss how He teaches or suggests the element of 
"monitoring feedback from the user as to which of the first and second images are 
relevant to the quer/*, but limits his discussion as to He's user feedback regarding 
modifying scores from marketing and engineering groups. 

He does not teach or suggest "identifying, in response to a search query 
second multimedia objects that have content features similar to those of the first 
multimedia objects/* Since neither Ono nor He suggest the ability for "identifying 
in response to a search query ... second multimedia objects that have content 
features similar to those of the first multimedia objects"* Claim 1 is allowable over 
the cited combination. Applicants respectfully request that the §103 rejection of 
claim 1 be withdrawn. 

Dependent claims 2-9 are allowable by virtue of their dependency on base 
claim L Applicants respectfully request that the §103 rejection of clainos 2-9 be 
withdrawn. 

Claims 10-11. 13-26. 27. 29-34, and 37-42 

Claims 10-11, 13-26, 28-34, and 37-42 are rejected under 35 U.S-C §103(a) 
as being unpatentable over He. Applicants respectfully traverse the rejection. 
Independent claim 10 recites: 

A method comprising: 

iteratively retrieving multimedia objects from a database and 
monitoring feedback from a user as to whether the multimedia objects arc 
relevant to a keyword in a search query; and 

annotating the multimedia objects based on the user's feedback, with 
the keyword. 
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1 He describes a system that provides multimedia content to iisers as well as 

2 summaries of the multimedia content. The summaries are generated to include 

3 portions of the multimedia content that other (previous) users may have found to 

4 be most interesting as identified flirough feedback by the other (previous) users. 

5 He, col 2 lines 20.27* 

6 The feedback described in He involves the use of "scores" attached to 

7 particular segments of the multimedia objects. The scores may be adjusted up or 

8 down by the users. Summaries show segments with the highest scores. He, col. 2, 
? lines 42-52. 

10 Annotations may be provided to the multimedia content where annotations 

11 include audio, text, video, etc. and generally provide a user's input such as a 

12 comment, question, or elaboration as to the multimedia content, Annotations 

13 typically correspond to a particularly temporal (time) location or range. He 

14 describes that these annotations are stored in a separate annotation server. A user 

15 communicates directly with the annotation server to add or retrieve annotations. 

16 He coL 3, line 67 to coL 4, line 16. 

17 Claim 10 recites "iteratively retrieving multimedia objects from a database 

18 and monitoring feedback from a user as to whether the multimedia objects are 

19 relevant to a keyword in a search query". 

20 The feedback described in He involves the use of "scores" that are attached 

21 by users to particular segments of the multimedia objects. The scores may be 
12 adjusted up or down by the users. He does not teach or suggest that such scores 

23 that are provided as user feedback are relevant to a keyword in a search query for 

24 multimedia objects. 



25 



16 



PAGE 1«25'RCVDAT Wm 3:19:40PM [Eastern Standard rone] *SVI^USPrO{FXRF-1/rDraS:8729306'CSID:509 323 B979' DURATION M 



-^^E5 PLL 509 323 8979 TO 17038^^3 



FEB 27 2004 12:32 FR LEE - RRYE5 PLL 509 323 8979 TO 17038^305 P. 19/25 



U 



1 The Examiner argues that He teaches '"monitoring feedback from a user as 

2 to whether the multimedia objects are relevant to a keyword in a search query". 

3 The feedback taught in He is directed to users adjusting scores assigned to 

4 segments of multimedia content. The scores of the user feedback in He are not 

5 directed keywords used in a search query for multimedia objects. 

6 Claim 10 further recites "annotating the multimedia objects based on the 

7 user's feedback, with the keyword", 

s He describes that multimedia content that may be annotated include audio, 

9 text, video^ etc, and generally provide a user's input such as a comment, question, 

10 or elaboration as to the multimedia content. Annotations may also correspond to a 
pardcularly temporal (time) location or range. The user conmnmicates directly 

12 with an annotation server, such that feedback provided by the user in the form of 

13 scores is separate from the annotation. 

14 The Examiner argues that He teaches "annotating the multimedia objects 

15 based on the user*s feedback, with the keyword". 

16 The annotation shown in He is directed in general to users providing a note 

17 (e.g., question, comment, elaboration, etc.) to multimedia contents Annotation 

18 with a keyword is not taught or suggested by He. Furthermore, since the user in 

19 He communicates directly with an annotation server, the annotation described in 

20 He does not depend or is related to the feedback of scores that are provided by the 

21 user. In other words, the annotation of He is not based on the user's feedback. 

22 Claim 10 is allowable over the cited reference. Applicants respectfully 

23 request that the §103 rejection of claim 10 be withdrawn. 



24 
2$ 



17 



PA(£ ' RCVD AT 2/2712004 3:19:40 PM [Eastern Standard Time] ' SVR:USPTO{^^^ 



^^^ES PLL 509 323 8979 TO 17038^^3 



FEB 27 2004 12=33 FR LEE - HRYES PLL 509 323 8979 TO 17038729306 P. 20/25 



1 

1 
3 
4 
5 
6 
7 
8 
9 
10 
11 
U 
13 
14 
15 
16 
17 
IS 
19 
20 
21 
22 
23 
24 
25 



Dependent claims 11, 13-17 are allowable by virtue of their dependency on 
base claim 10. Applicants respectfully request that the §103 rejection of claims 1 1, 
13-17 be withdrawn. 

Independent claim 18 recites: 

A method comprising: 

retrieving multimedia objects according to a content-based retrieval 
process; 

presenting the multimedia objects to a user; 

monitoring feedback from the user as to which of the multimedia 
objects are relevant; and 

annotating one or more of the multimedia objects based on the user's 
feedback, with a keyword. 

He does not teach or suggest the element of "monitoring feedback from the 
user as to which of the multimedia objects are relevant". As discussed above in 
support of claim 10, the feedback taught in He is directed to adjusting a score to a 
segment of a multimedia content. 

He does not teach or suggest the element of "annotating one or more of the 
multimedia objects based on the user*s feedback, with a keyword". As discussed 
above in support of claim 10, the feedback of He is not directed to the annotation 
of He. In other words the feedback and the annotation shown in He are separate 
from another, and the annotation does not make use of the feedback. Furthermore, 
as discussed above, He does not teach or suggest the use of a keyword in 
annotation. 

Claim 18 is allowable over the cited reference. Applicants respectfully 
request that the § 103 rejection of claim 18 be withdrawn. 
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Dependent claims 19-24 are allowable by virtue of their dependency on 
base claim 10. Applicants respectfiilly request that the §103 rejection of claims 
19-24 be withdrawn. 

Independent claim 25 recites: 

A method comprising: 

maintaining associations between keywords and multimedia objects, 
the associations being weighted to indicate how relevant the keywords are 
to the multimedia objects; 

retrieving a set of one or more multimedia objects for presentation to 

a user; 

monitoring feedback from the user as to which of the multimedia 
objects are relevant; and 

adjusting the weights of the associations based on the user's 
feedback. 

He does not teach or suggest the element of ^^maintaining associations 
between keywords and multimedia objects, the associations being weighted to 
indicate how relevant the keywords are to the multimedia objects". 

The Examiner argues that one group of users such a marketing group may 
modify segments scores of another user or group of users such as an engineering 
group. Applicants do not disagree with the Examiner*s interpretation of He; 
however, Applicants disagree witii the Examiner that this association between the 
groups (users) corresponds to multimedia objects and keywords. As discussed He 
shows that segments of multimedia content may be provided a score that as the 
Examiner has pointed may be changed by one or more groups (users). He does not 
teach or suggest associations between keywords and multimedia objects. Weights 
are taught by He's scores associated with segments of multimedia content; 
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however, He does not teach or suggest that weights (scores) may be assigned to 
keywords as to their relevance to multimedia objects. 

Claim 25 is allowable over the cited reference. Applicants respectfully 
4 request that the § 103 rejection of claim 25 be withdrawn, 

s Dependent claims 26 and 28-31 are allowable by virtue of their 

6 dependency on base claim 25, Applicants respectfully request that the §103 

7 rejection of claims 26 and 28-3 1 be withdrawn. 

8 Independent claim 32 recites: 



9 



10 



A system comprising: 



an information retrieval unit to retrieve multimedia objects from a 

1 1 database based on a search query; 

^2 a relevance feedback unit to capture a user's feedback as to whether 

J J the multimedia objects are relevant to the search query; and 

14 an annotation unit to annotate, with a keyword, the multimedia 
objects based on the user's feedback. 

IS 

16 1 As discussed above, the feedback taught in He is directed to providing a 

17 score as to segments of multimedia content* He does not teach or suggest that such 

18 feedback be used as to relevance of the segments of the mxiltimedia content 
^9 (multimedia content) to a search query. Therefore He does not teach or suggest the 

20 element of "a relevance feedback imit to capture a user*s feedback as to whether 

21 the multimedia objects are relevant to the search quer/\ 

22 Furthemiore, as discussed above, the feedback of He is not directed to the 

23 annotation of He. The feedback and the annotation shown in He are separate from 
2* another, and the annotation does not make use of the feedback. Therefore He does 
25 not teach or suggest the element of "an annotation unit to annotate, with a 
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keyword, the multimedia objects based on the user's feedback'*. Also as discussed 
above, He does not teach or suggest the use of a keyword in annotation. 

Claim 25 is allowable over the cited reference. Applicants respectfully 
request that the § 103 rejection of claim 25 be withdrawn. 

Dependent claims 33-34 and 37-42 are allowable by virtue of their 
dependency on base claim 32, Applicants respectfully request that the §103 
rejection of claims 33-34 and 37-42 be withdrawn. 

Claims 12, 27. and 35-36 

Claims 12, 27 and 35-36 are rejected under 35 U.S.C. § 103(a) as being 
unpatentable over He in view of Ono, Applicants respectfully traverse the 
rqection. 

Dependent claim 12 depends from claim 10 and hence incorporates the 
features of claim 10. As such claim 12 requires "iteratively retrieving multimedia 
objects from a database and monitoring feedback from a user as to whether the 
multimedia objects are relevant to a keyword in a search query; and annotating the 
multimedia objects based on the user's feedback, with the keyword*'. 

Ono is cited for teaching "the searching utilizing both keywords and 
features". However, Ono provides no assistance in light of He as to the recited 
method of claim 10 from which claim 12 depends. 

Applicants assert the arguments in support of claim 10. Applicants 
respectfully request that the §103 rqection of claim 12 be withdrawn. 

Dependent claims 35 and 36 depend from claim 32 and hence incorporate 
the features of claim 32, As such claims 35 and 36 requires "a relevance feedback 
unit to capture a user's feedback as to whether the multimedia objects are relevant 
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to the search query; and an annotation unit to annotate, with a keyword, the 
multimedia objects based on the user's feedback". 

Ono is cited for teaching '*the searching utilizing both keywords and 
features". However, Ono provides no assistance in light of He as to the recited 
method of claim 32 from which claims 35 and 36 depend. 

Applicants assert the arguments in support of claim 10. Applicants 
respectfully request that the §103 rejection of claims 35 and 36 be withdrawn. 
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CONCLUSION 

All pending claims 1-42 are in condition for allowance. Applicant 
respectfully requests reconsideration and prompt issuance of the subject 
application. If any issues remain that prevent issuance of this application, the 
Examiner is urged to contact the undersigned attorney before issuing a subsequent 
Action, 



Dated: 



Respectfully Submitted, 



By: 



Emmanuel Rivera 

Reg. No. 45,760 

(509) 324-9256 ext. 245 
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